Using AI Uncertainty Quantification to Improve Human Decision-Making

AI Uncertainty Quantification (UQ) has the potential to improve human decision-making beyond AI predictions alone by providing additional useful probabilistic information to users. The majority of past research on AI and human decision-making has concentrated on model explainability and interpretability. We implemented instance-based UQ for three real datasets. To achieve this, we trained different AI models for classification for each dataset, and used random samples generated around the neighborhood of the given instance to create confidence intervals for UQ. The computed UQ was calibrated using a strictly proper scoring rule as a form of quality assurance for UQ. We then conducted two preregistered online behavioral experiments that compared objective human decision-making performance under different AI information conditions, including UQ. In Experiment 1, we compared decision-making for no AI (control), AI prediction alone, and AI prediction with a visualization of UQ. We found UQ significantly improved decision-making beyond the other two conditions. In Experiment 2, we focused on comparing different representations of UQ information: Point vs. distribution of uncertainty and visualization type (needle vs. dotplot). We did not find meaningful differences in decision-making performance among these different representations of UQ. Overall, our results indicate that human decision-making can be improved by providing UQ information along with AI predictions, and that this benefit generalizes across a variety of representations of UQ.Comment: 10 pages and 7 figure

Repeated Measures Correlation

Repeated measures correlation (rmcorr) is a statistical technique for determining the common within-individual association for paired measures assessed on two or more occasions for multiple individuals. Simple regression/correlation is often applied to non-independent observations or aggregated data; this may produce biased, specious results due to violation of independence and/or differing patterns between-participants versus within-participants. Unlike simple regression/correlation, rmcorr does not violate the assumption of independence of observations. Also, rmcorr tends to have much greater statistical power because neither averaging nor aggregation is necessary for an intra-individual research question. Rmcorr estimates the common regression slope, the association shared among individuals. To make rmcorr accessible, we provide background information for its assumptions and equations, visualization, power, and tradeoffs with rmcorr compared to multilevel modeling. We introduce the R package (rmcorr) and demonstrate its use for inferential statistics and visualization with two example datasets. The examples are used to illustrate research questions at different levels of analysis, intra-individual, and inter-individual. Rmcorr is well-suited for research questions regarding the common linear association in paired repeated measures data. All results are fully reproducible

Malware in the Future? Forecasting of Analyst Detection of Cyber Events

There have been extensive efforts in government, academia, and industry to anticipate, forecast, and mitigate cyber attacks. A common approach is time-series forecasting of cyber attacks based on data from network telescopes, honeypots, and automated intrusion detection/prevention systems. This research has uncovered key insights such as systematicity in cyber attacks. Here, we propose an alternate perspective of this problem by performing forecasting of attacks that are analyst-detected and -verified occurrences of malware. We call these instances of malware cyber event data. Specifically, our dataset was analyst-detected incidents from a large operational Computer Security Service Provider (CSSP) for the U.S. Department of Defense, which rarely relies only on automated systems. Our data set consists of weekly counts of cyber events over approximately seven years. Since all cyber events were validated by analysts, our dataset is unlikely to have false positives which are often endemic in other sources of data. Further, the higher-quality data could be used for a number for resource allocation, estimation of security resources, and the development of effective risk-management strategies. We used a Bayesian State Space Model for forecasting and found that events one week ahead could be predicted. To quantify bursts, we used a Markov model. Our findings of systematicity in analyst-detected cyber attacks are consistent with previous work using other sources. The advanced information provided by a forecast may help with threat awareness by providing a probable value and range for future cyber events one week ahead. Other potential applications for cyber event forecasting include proactive allocation of resources and capabilities for cyber defense (e.g., analyst staffing and sensor configuration) in CSSPs. Enhanced threat awareness may improve cybersecurity.Comment: Revised version resubmitted to journa

Does Explainable Artificial Intelligence Improve Human Decision-Making?

Explainable AI provides insight into the "why" for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. Whether explainable AI can improve actual human decision-making and the ability to identify the problems with the underlying model are open questions. Using real datasets, we compare and evaluate objective human decision accuracy without AI (control), with an AI prediction (no explanation), and AI prediction with explanation. We find providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact. Moreover, we observed the strongest predictor for human decision accuracy was AI accuracy and that users were somewhat able to detect when the AI was correct versus incorrect, but this was not significantly affected by including an explanation. Our results indicate that, at least in some situations, the "why" information provided in explainable AI may not enhance user decision-making, and further research may be needed to understand how to integrate explainable AI into real systems

Pharmacology of MDMA- and Amphetamine-Like New Psychoactive Substances

New psychoactive substances (NPS) with amphetamine-, aminoindan-, and benzofuran basic chemical structures have recently emerged for recreational drug use. Detailed information about their psychotropic effects and health risks is often limited. At the same time, it emerged that the pharmacological profiles of these NPS resemble those of amphetamine or 3,4-methylenedioxymethamphetamine (MDMA). Amphetamine-like NPS induce psychostimulation and euphoria mediated predominantly by norepinephrine (NE) and dopamine (DA) transporter (NET and DAT) inhibition and transporter-mediated release of NE and DA, thus showing a more catecholamine-selective profile. MDMA-like NPS frequently induce well-being, empathy, and prosocial effects and have only moderate psychostimulant properties. These MDMA-like substances primarily act by inhibiting the serotonin (5-HT) transporter (SERT) and NET, also inducing 5-HT and NE release. Monoamine receptor interactions vary considerably among amphetamine- and MDMA-like NPS. Clinically, amphetamine- and MDMA-like NPS can induce sympathomimetic toxicity. The aim of this chapter is to review the state of knowledge regarding these substances with a focus on the description of the in vitro pharmacology of selected amphetamine- and MDMA-like NPS. In addition, it is aimed to provide links between pharmacological profiles and in vivo effects and toxicity, which leads to the conclusion that abuse liability for amphetamine-like NPS may be higher than for MDMA-like NPS, but that the risk for developing the life-threatening serotonin syndrome may be increased for MDMA-like NPS

Mission command in the age of network-enabled operations: social network analysis of information sharing and situation awareness

The article of record as published may be found at http://dx.doi.org/10.3389/fpsyg.2016.00937Reviewed by: Sean Everton, Naval Postgraduate School, USA Susan L. McDonald, SAIC USA, USAA common assumption in organizations is that information sharing improves situation awareness and ultimately organizational effectiveness. The sheer volume and rapid pace of information and communications received and readily accessible through computer networks, however, can overwhelm individuals, resulting in data overload from a combination of diverse data sources, multiple data formats, and large data volumes. The current conceptual framework of network enabled operations (NEO) posits that robust networking and information sharing act as a positive feedback loop resulting in greater situation awareness and mission effectiveness in military operations (Alberts and Garstka, 2004). We test this assumption in a large-scale, 2-week military training exercise. We conducted a social network analysis of email communications among the multi-echelon Mission Command staff (one Division and two sub-ordinate Brigades) and assessed the situational awareness of every individual. Results from our exponential random graph models challenge the aforementioned assumption, as increased email output was associated with lower individual situation awareness. It emerged that higher situation awareness was associated with a lower probability of out-ties, so that broadly sending many messages decreased the likelihood of attaining situation awareness. This challenges the hypothesis that increased information sharing improves situation awareness, at least for those doing the bulk of the sharing. In addition, we observed two trends that reflect a compartmentalizing of networked information sharing as email links l were more commonly formed among members of the command staff with both similar functions and levels of situation awareness, than between two individuals with dissimilar functions and levels of situation awareness; both those findings can be interpreted to reflect effects of homophily. Our results have major implications that challenge the current conceptual framework of NEO. In addition, the information sharing network was largely imbalanced and dominated by a few key individuals so that most individuals in the network have very few email connections, but a small number of individuals have very many connections. These results highlight several major growing pains for networked organizations and military organizations in particular.Network Science Collaborative Technology Alliance (USARL)Cooperative Agreement no. W911NF-09-2-005